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SUMMARY 


Computer vision research has led to the development of several algorithms for estimating range 
to obstacles during low-altitude flight. However, due to the limited availability of “real world” data, 
algorithm verification has not been effectively addressed. A helicopter flight test experiment has been 
conducted at NASA Ames Research Center to obtain a database consisting of video imagery and 
accurate measurements of camera motion, camera calibration parameters, and true range information. 
The database was developed to allow verification of monocular passive range estimation algorithms for 
use in the autonomous navigation of rotorcraft during low altitude flight. This paper briefly describes 
the helicopter flight experiment and presents four data sets representative of the different helicopter 
maneuvers and the visual scenery encountered during the flight test. These data sets will be made 
available to researchers in the computer vision community. 

INTRODUCTION 

NASA, in conjunction with the US Army, has been pursuing research in autonomous navigation of 
rotorcraft during low-altitude flight in order to reduce the high pilot workload associated with obstacle 
avoidance (ref. 1). In one approach, obstacle information is acquired by a computer vision system 
located on board the rotorcraft. The obstacle information would be used to generate advisory displays 
and serve as an input to an automatic guidance system capable of performing the obstacle avoidance 
task autonomously. Research at Ames (refs. 2-4) has focused on the development of range estimation 
algorithms and obstacle avoidance algorithms for autonomous navigation (ref. 5). Since military he- 
licopters are increasingly being equipped with inertial systems to measure the vehicle’s motion states 
for guidance and stability augmentation systems, the range estimation algorithms use knowledge of the 
camera’s motion (position, orientation, linear velocity, and angular velocity) and estimate the range to 
environmental points. 

Experimental data are needed to establish the validity of algorithms and to investigate factors 
encountered in real world data that affect algorithmic performance. A laboratory facility has been 
developed at Ames to provide the first stage of algorithm verification (ref. 6). Further development 
and testing of these algorithms require data collected from rotorcraft flight. Current databases available 
in the literature contain some motion information and true range measurements for outdoor scenery 
(refs. 7-9) but they either do not provide extensive motion measurements or they do not describe the 
general camera motion (translation and rotation) encountered in flight. A flight experiment has been 
conducted to obtain the video imagery, camera motion, camera calibration parameters, and true range 
information for testing algorithms with realistic data. It is hoped that availability of these data sets will 
facilitate the comparison of different motion analysis methods. 

DATA SET DEVELOPMENT 

Figure 1 depicts the flight experiment designed to acquire the necessary measurements of rotorcraft 
motion states, video imagery, and true range information. A Cohu 6410-series monochrome CCD video 
camera was mounted rigidly under the nose of a CH-47B Chinook helicopter and oriented approximately 
along the longitudinal body axis to observe obstacles that the rotorcraft would encounter in forward 




Figure 1 . Right experiment overview. 


flight. The camera produces standard RS-170 interlaced video output and has an electronic shutter which 
was set for 1 msec exposure time (per video field). The analog video signal was recorded by a U-matic 
video tape recorder located onboard the CH-47. 

The rotorcraft motion states were measured by instrumentation on board the CH-47, filtered, dig- 
itized, and transmitted to a ground station facility for recording. Raw measurements including linear 
accelerations, Euler angles, and angular rates were collected. The instruments were aligned with the 
helicopter’s body axes which originate at the helicopter’s center of gravity. The body axes system and 
other coordinate systems of interest are illustrated in figure 2. Derivation of the camera motion states 
from the measured rotorcraft motion states requires knowledge of the position and orientation of the 
camera axes system relative to the helicopter body axes system, as will be addressed later in this section. 
The motion measurements have a minimum bandwidth of 10 Hz and were sampled at approximately 
110 Hz. The rotational frequency of the helicopter’s rotor blades is about 11 Hz. 

True range measurements were obtained by a two-step process using a laser tracker. First, the 
laser tracker measured the position in Earth axes of a rotorcraft-mounted reflector throughout each test 
flight, and the resulting data (also at 110 Hz) were recorded at the ground station on a common time 
base with the telemetry data. Second, at the completion of a test flight, the laser tracker was used to 
measure the position in Earth axes of the (stationary) objects that served as the obstacles of interest. A 
laser reflector was manually placed at each obstacle location to obtain the position information. 

To coordinate the imagery data with the rotorcraft state data and the true range measurements, a 
time source onboard the CH-47 was synchronized with the time source at the ground station to 1-msec 
accuracy. A message containing the current time was then displayed in the upper left-hand comer of 
each video image. 

Significant post-flight processing of the data was required to develop the raw measurements into a 
final form suitable for motion analysis research. An overview of the post-flight processing procedure is 
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shown in figure 3. Helicopter motion measurements were corrected to the CG-centered body axes using 
instrument position information. The location in body axes of the laser reflector was used along with 
helicopter orientation information to determine the helicopter CG location in Earth axes from the raw 
laser measurements. All measurements were then low-pass filtered at 10 Hz before eventual subsampling 
to 30 Hz video rate. 

The motion states were processed using a state estimation algorithm (ref. 10) to check the accu- 
racy and kinematic consistency of the measurements. This algorithm uses the well-known rigid body 
kinematic equations of motion to process the measurements in an “optimal” way to ensure internal con- 
sistency of measured states, improve knowledge of poorly measured states, identify instrument bias and 
scale factor errors, and estimate states during periods of telemetry dropout. Since direct measurements 
of linear velocity were unavailable, velocity information was reconstructed by state estimation based 
upon position and acceleration measurements. The end result is a single best estimate of the rotorcraft 
motion states (position, orientation, linear velocity, angular rates, and linear accelerations) based on all 
available measurements. The motion states are consistent with instrument accuracy for frequencies up 
to 15 Hz. State estimation processing was used to develop a high-precision, internally consistent data 
set for research purposes; however, for operational systems the state measurement information required 
for passive range estimation would be acquired directly from onboard instrumentation. 

Knowledge of the position and orientation of the camera axes system with respect to the helicopter 
body axes system is necessary to derive the camera motion states (in camera axes) given the rotorcraft 
motion states (in body axes). The camera axes originate at the camera lens focal point with the X s 
axis oriented along the optical axis and the Y s and Z s axes oriented along the rows and columns, 
respectively, of the sensor array (see fig. 2). The position and orientation of the camera axes are known 
as the external camera calibration parameters. Additional information about the camera such as the 
focal length, the location of the image center (where the optical axis passes through the image plane). 
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Figure 3. Development of data set elements. 


and the effective pixel aspect ratio (known collectively as the internal camera calibration parameters) is 
also necessary for motion analysis. 

It is difficult to directly measure even the external parameters with sufficient accuracy, so both the 
internal and external parameters are determined experimentally. The external parameters map points 
from body axes to camera axes and the internal parameters map points from camera axes to locations on 
the image plane. By measuring the location of points in body axes and the corresponding pixel location 
where the points appears in an image, it is possible to estimate the internal and external parameters. 
While conceptually straightforward, the measurement in body axes of points in the camera’s field of 
view is a challenging experimental task. The method used to collect the experimental data and the 
technique developed to estimate the unknown parameters are discussed in reference 11. 


DATA SET CONTENTS 


The flight experiment and processing of the raw data results in the following information: 

1. digitized imagery (30 frames/sec) 

2. sensor velocity and angular rates in camera axes 

3. sensor position and orientation in Earth axes 
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4. rotorcraft velocity and angular rates in body axes 

5. rotorcraft position and orientation in Earth axes 

6. internal camera calibration parameters 

7. external camera calibration parameters 

8. obstacle positions in Earth axes 


The digitized images are 512 x 512 pixels with 256 gray levels. The rotorcraft and camera motion 
measurements corresponding to each image as well as the camera calibration parameters are stored in 
a header that prefaces each image. Specific remarks concerning use of the image header contents may 
be found in the appendix. The obstacle position measurements in Earth axes are provided in a separate 
file. Each image header contains the information necessary to express obstacle positions in terms of 
the helicopter body axes, camera axes, or image plane pixel coordinates. Figures 4-7 show sample 
images collected during the flight experiment. The object visible in the upper right-hand comer of each 
image is the helicopter’s nose boom. The motion and calibration information for the left-hand image in 
figure 4 is given in table 1. The image header contains the data in the first two columns of the table. 
The information concerning accuracy and units is the same for all image headers. Table 2 contains the 
position measurements in Earth axes for the labelled points in figures 4 and 5. 



Figure 4. First and last images of Line data set. 
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Table 1. Sample image header data 


Measurement name 

Value 

Accuracy 

Units 

SENSOR_POSITION_X_WORLD 

734 


feet 

SENSOR_POSITION_Y .WORLD 

520 


feet 

SENSOR .POSITION _Z_W ORLD 

-11 


feet 

BODY POSITION _X_W ORLD 

757 

2.0 

feet 

BODY-POSITION-Y -WORLD 

517 

2.0 

feet 

BODY -POSITION-Z-WORLD 

-16 

2.0 

feet 

SENSOR. VELOCITY_X -SENSOR 

30.2 


feet/sec 

SENSOR. VELOCITY_Y -SENSOR 

0.2 


feet/sec 

SEN SOR_ VELOCITY _Z_SEN SOR 

-1.9 


feet/sec 

SENSOR.ANGUL AR_RATE_X_SEN S OR 

0.0238 


rad/sec 

SENSOR-ANGULARJRATE.Y .SENSOR 

0.0113 


rad/sec 

SENSOR-ANGULAR -RATE -Z.SENSOR 

0.0124 


rad/sec 

BODY-VELOCITY _X_BODY 

30.1 

0.3 

feet/sec 

BODY_VELOCITY_Y -BODY 

0.1 

0.3 

feet/sec 

BODY_VELOCITY_Z_BODY 

2.6 

0.6 

feet/sec 

BODY -ANGULAR-RATE-X .BODY 

0.0218 

0.0045 

rad/sec 

BODY _ANGULAR_RATE_Y -BODY 

0.0115 

0.0045 

rad/sec 

BODY_ANGULAR_RATE JZ-BODY 

0.0155 

0.0025 

rad/sec 

SENSOR_POSITION_X_BODY 

22.950 

0.042 

feet 

SENSOR-POSITION-Y -BODY 

-1.043 

0.017 

feet 

SENSOR_POSmON_Z_BODY 

6.940 

0.017 

feet 

ANGLE_PSI_WORLD_TO_BODY 

3.0348 

0.0123 

radians 

ANGLE_THETA_WORLD.TO.BODY 

0.0646 

0.0021 

radians 

ANGLE_PHI_WORLD_TO_BODY 

-0.0153 

0.0042 

radians 

ANGLE_PSI_BODY_TO-SENSOR 

0.0055 

0.0035 

radians 

ANGLE_THETA_BODY_TO_SENSOR 

-0.1393 

0.0035 

radians 

ANGLE_PHI_BODY_TO_SENSOR 

-0.0074 

0.0017 

radians 

ANGLE_PSI_WORLD_TO_SENSOR 

3.0424 


radians 

ANGLE-THETA-WORLD.TO-SENSOR 

-0.0746 


radians 

ANGLE_PHI_WORLD .TO-SENSOR 

-0.0223 


radians 

ASPECT-RATIO 

1.005 

0.001 

non-dimensional 

FOCALXENGTH 

621.4 

2.6 

pixels 

U.CENTER 

253.3 

2.4 

pixels 

V-CENTER 

238.3 

1.6 

pixels 

STAMP-TIME 

236:22:31:31.061 


seconds 

GLOBAL-TIME 

81091.061 


seconds 

DELTA-TIME 

0.033 


seconds 

FRAMEID 

0 


non-dimensional 
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Table 2. Sample true obstacle position data 


Label 

Obstacle name 

X e , ft 

Y e , ft 

Z e , ft 

A 

truck 2, NE ground level 

368.7 ± 2 

614.1 ± 2 

4.5 ± 2 

B 

truck 2, SE top comer 

348.7 

614.5 

-3.1 

C 

truck 4, NE ground level 

118.3 

633.1 

3.6 

D 

truck 4, SE top comer 

98.6 

634.8 

-7.4 

E 

truck 5, NE ground level 

-17.7 

510.6 

3.0 

F 

truck 5, SE top comer 

-37.6 

511.5 

-7.7 

G 

truck 3, NE ground level 

231.2 

490.2 

3.9 

H 

truck 3, SE top comer 

209.0 

491.9 

-0.2 

I 

truck 1, NE ground level 

479.3 

470.6 

4.9 

J 

truck 1, SE top comer 

461.5 

472.3 

-3.0 

K 

truck 1, mast tip 

460.0 

466.7 

-24.3 



Figure 5. First and last images of Arc data set. 
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Figure 6. First and last images of Posts data set. 



Figure 7. First and last images of Towers data set. 
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EXAMPLE DATA SETS 


Each of the following data sets was selected to demonstrate practical or operational tasks for 
a helicopter during low-altitude flight. The data sets represent various levels of difficulty for range 
estimation algorithms. Each data set consists of 90 images and the associated supporting data described 
in the previous section. The first and last images of each data set are shown in figures 4-7. White dots 
in the figures indicate obstacles whose position has been measured. Since the laser tracker and most 
obstacles of interest are permanently fixed in position, a greater quantity of obstacle position information 
may be available in the future. 


Line Data Set 

The Line data set was acquired as the CH-47 flew a straight-line path between two rows of vehicles 
stationed along a runway. This data set demonstrates the deviation from theoretical straight-line motion 
that may be encountered under operational conditions for helicopter flight. The Line data set was 
designed to be simple both in terms of camera motion and scenery, and is therefore well suited for 
testing range estimation algorithms using flight data. The camera velocity is roughly 30 ft/sec along 
the optical axis, giving a camera motion of about 1 ft between successive images. Velocity components 
orthogonal to the optical axis change as much as 1.5 ft/sec over the 3-second period. Angular rates up 
to 0.05 rad/sec and changes in orientation of 0.03 rad are observed. The location of the lower-front 
and upper-rear comers on the right hand side of each obstacle was measured to provide true range 
information as well as data on the size of the obstacles. Range to the obstacles varies from 200 to 
800 ft. The truck on the far right of figure 4 has an extensible 20 ft tower whose location was also 
determined. 


Arc Data Set 

The Arc data set uses the same simplified scenery as in the Line data set, but complexity of the 
camera motion is increased by having the helicopter follow an S-shaped ground path. The Arc data set 
captures one curve from that flight profile. The Arc sequence allows testing with simplified imagery of 
range estimation algorithms designed to operate with generalized camera motion. Peak angular velocity 
in yaw is about 0.13 rad/sec and is maintained for 1 sec. A bank angle of 0.045 rad is attained during 
the turn. Velocity along the camera’s optical axis is roughly 40 ft/sec. The obstacles are 200 to 650 ft 
from the camera. 


Posts Data Set 

The Posts data set demonstrates straight-line motion but with imagery which contains both manmade 
and natural objects. The availability of distinct objects in the imagery (for example, the white posts) 
facilitates the validation of range estimates. The camera has a velocity of roughly 40 ft/sec along the 
optical axis. Distinctive objects are between 80 and 350 ft from the camera. 
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Towers Data Set 


The Towers data set combines straight-line motion with very challenging imagery. An effective 
operational range estimation system should be able to successfully process data of this nature. The 
camera is moving at a velocity of 90 ft/sec along its optical axis. The nearest transmission towers are 
at a range of at least 450 ft. 


SUMMARY AND CONCLUSIONS 

A database has been developed based on a helicopter flight test experiment to allow validation 
of passive range estimation algorithms with realistic camera motion and visual scenery. The database 
includes video imagery, measurements of camera motion, and information on the characteristic param- 
eters of the camera. In addition, independent measurements of range are included to allow verification 
of range estimates. 

Four data sets from the larger database have been presented here. These data sets represent various 
camera motions and visual scenery, and were selected to provide a sequence of increasingly challenging 
tests for passive range estimation algorithms. These data sets will be available to researchers in the 
computer vision community. 

Future plans include the development of a database to support multicamera methods of passive 
range estimation. Additional efforts may include the collection of infrared imagery to investigate the 
feasibility of performing range estimation at night. 
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APPENDIX 


A few remarks concerning the image header data are in order to facilitate use of the data sets. It 
is noted that the image and header data for each sample time are provided in a single file stored in a 
HIPS-compatible format. A subroutine (written in the C programming language) which can access the 
image and header information will be provided with the data sets. Specific comments on the use of the 
header data follow. 

Coordinate Systems 

The coordinate systems used to express measurements contained in the data sets are illustrated ; n 
figure 2 and are defined below: 

1. Earth (world) frame - The Earth frame (also called the world frame in the image headers) is 
rigidly affixed to the Earth with the X e axis pointing North, the Y e axis pointing East, and the Z e axis 
pointing toward the center of the Earth. The origin of the Earth frame is an arbitrarily selected point 
on a runway at the test flight facility. 

2 . Helicopter body frame - The helicopter body axes frame (or body frame) is assumed to be 
fixed relative to the helicopter with the X 5 axis pointing forward out the helicopter nose, the axis 
pointing out the right hand side of the helicopter, and the Z f, axis pointing downward relative to the 
helicopter’s geometry (i.e., not necessarily toward the center of the Earth). The origin of the body frame 
is the helicopter’s nominal center of gravity. 

3. Image Plane Axes - The image plane axes are oriented along the rows and columns of the 
sensor array. The u axis points to the right along rows and the v axis points downward along the 
columns. The image plane axes originate at the image center (i.e., where the optical axis passes through 
the image plane). 

4. Camera (sensor) frame - The camera frame is rigidly attached to the camera and originates at 
the lens focal point. The Y s and Z s axes are parallel to the image plane axes u and v, respectively. 
The X s axis points along the optical axis. Since the camera is rigidly mounted to the helicopter, the 
location and orientation of the camera frame remains constant in body axes. 

5. Pixel axes — The pixel axes, n u and n v , are attached to the camera’s image plane and point 
along the rows and columns of the sensor array as do the image plane axes; however, the pixel axes 
originate at the upper left-hand comer of the sensor array rather than at the image center. In addition, 
distances along these axes are expressed in units of pixels (which are not necessarily square), so the 
coordinates of any point are its row and column indices in the image array. The upper left-hand pixel 
has coordinates ( 0 , 0 ). 

Naming Convention 

The naming convention for motion variables contained in the image headers is defined below: 

1. Position, velocity, and angular rate measurements: 

a. The first term names a coordinate frame whose motion is to be given (i.e., SENSOR or 
BODY). 
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b. The second term indicates a motion state (i.e., POSITION, VELOCITY, or ANGULAR _RATE). 

c. The final term specifies a component of the motion state (i.e., X .WORLD, YJBODY, 
Z_SENSOR, etc.). 

Example: SENSOR_POSITION_X_WORLD 
Interpretation: component of sensor position along the X e axis 
2. Angle measurements: 

a. The first term is always ANGLE. 

b. The second term indicates the Euler angle to be specified (i.e., PSI, THETA, or PHI). See 
the following section, Rotation Matrices, for definition of the Euler angles. 

c. The final term indicates the base coordinate frame and the destination coordinate frame (i.e., 
WORLD _TO_BODY, BODY_TO_SENSOR, WORLD.TO.SENSOR). 

Example: ANGLE_PHI_WORLD_TO_BODY 

Interpretation: bank angle of the body frame relative to the world frame 


Rotation Matrices 


The rotation matrices are defined in terms of Euler angles. The Euler angles are heading angle ip, 
attitude angle 0, and bank angle <p. The rotation sequence is ip about Z, 6 about Y, and <p about X, 
where all rotations are positive in the right-hand sense. The rotation matrix resulting from this sequence 
of rotations is given below 


R{ip,0,<p) 


aped sipcQ —sd 

cips6s<p — sipap sipsOscp -I- cipccp c6s<p 
cipsOcip -|- sips(p sips6c<p — cips<p c6c<p 


The rotation matrix premultiplies a column vector to express that vector in another coordinate frame. 
The rotation matrices from the Earth frame to the body frame, from the body frame to the camera frame, 
and from the Earth frame to the camera frame are given by the following equations 


R-be — Rfybe > ^bei fibe) 

R sb = -RW’fifr) 6sb> 4>$b ) 

Rse — R s (fRbe — Ri^Pse, $se> <Pse) 

where Rf, e is the rotation matrix from the Earth frame to the body frame, ip ^ is ANGLE J > SI_WORLD 
_TO_BODY, etc. The Euler angles are given in the image headers. 


Perspective Projection Equations 


The pixels which compose the images are not square for reasons discussed in reference 11. The 
pixel aspect ratio is defined as follows 

b = 8v/6u 



where 6u and Sv are the horizontal and vertical pixel spacing, respectively. The effective focal length of 
the lens, f e , is expressed in units of vertical pixels. The origin of the image plane coordinates (i.e., the 
image center) is located at (nuQ,n Vo ) in the pixel axes system. With these definitions, the perspective 
projection equations which map points from the camera axes system to a pixel location in the image 
array are 


n u = n u o + bf e (y s /x s ) 


n v — n v o + f e {z s /xs) 


where (x s ,y s ,z s ) is the location of a point in camera axes and (n u ,n v ) is its projected location on the 
image plane in pixel axes. The following values are provided in the image header: b (ASPECT-RATIO ), 
fe (FOCAL-LENGTH), n UQ (ILCENTER), and n VQ (V-CENTER). The value of the vertical pixel 
spacing, 6v, is 3.89 x 10~ 4 inch. 
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